Repository navigation
perf: Density states in DuckDB's arena; take the library from the Query.Farm fork - #34
Merged
Merged
Conversation
…ry.Farm fork The density sketch's kernel only accepted std::vector<double>, so the sketch could not use DSAllocator and its aggregate states stayed on the heap, out of reach of memory_limit (3.9 GB under a 1 GB limit for 2M groups). The library now comes from the fork's duckdb-5.3.0 branch: release 5.3.0 plus one commit per fix sent upstream (apache/datasketches-cpp#552, #555-#562), replacing the overlay's patch files. #562 makes the density kernel take any vector. Density aggregates keep their sketches, points included, in the aggregate arena with export and import. Points copy with their own allocator, so input sketches are read into a scratch arena and merged while the state's arena is current. The same query now runs in 1.1 GB under 1 GB (7.6 s, from 10.1 s) and in 3.2 s under 4 GB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Density was the last sketch family whose aggregate states stayed on the heap. The library's kernel only accepted
std::vector<double>, so the sketch couldn't use our allocator.Library from the fork
vcpkg-overlay/apache-datasketchesnow pointsvcpkg_from_githubatrustyconover/datasketches-cpp, branchduckdb-5.3.0(44f777b). That branch is release 5.3.0 plus one commit per fix we've sent upstream, and it replaces the overlay's 8 patch files:merge(const&)called an unqualified swapstd::vector<T>, and an unused allocator member was always default-constructedThe fork branch passes all 17 library test suites. When an upstream release includes a fix, its commit can be dropped and the REF moved.
Density state
Density aggregates now keep their sketches, points included, in the aggregate arena, and export and import them. A point copies with its own allocator, so:
Tests
New
test/sql/datasketch_spill_density.test, under 128 MB with 4 threads:EXPORT_STATEround tripsAll tests pass (37 test files, 2165 assertions).
🤖 Generated with Claude Code